Dynamic adjustments in mixed-reality environment based on positional assumptions and configurations

ABSTRACT

Systems and methods are provided for generating, selecting, modifying and/or otherwise configuring occluder(s) used in mixed-reality environments. Pose information of a trackable device worn or carried by a user is determined. Based on that pose information, an assumption regarding the user&#39;s own pose is determined. An occluder is then configured in a manner to correspond to the assumed pose of the user for use in the mixed-reality environment. By generating the occluder in this manner, improved battery life by the trackable device can be achieved relative to conventional systems that rely entirely on iterative scanning of the user&#39;s body to determine body position.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.16/932014 filed on Jul. 17, 2020, entitled “DYNAMIC ADJUSTMENTS INMIXED-REALITY ENVIRONMENT BASED ON POSITIONAL ASSUMPTIONS ANDCONFIGURATIONS,” which issued as U.S. Pat. No. 11,354,866 on Jun. 7,2022, and which application is expressly incorporated herein byreference in its entirety.

BACKGROUND

Mixed-reality (MR) systems, including virtual-reality (VR) andaugmented-reality (AR) systems, have received significant attentionbecause of their ability to create truly unique experiences for theirusers. For reference, conventional VR systems create completelyimmersive experiences by restricting their users' views to only virtualenvironments. This is often achieved through the use of a head mounteddevice (HMD) that completely blocks any view of the real world. As aresult, a user is entirely immersed within the virtual environment. Incontrast, conventional AR systems create an augmented-reality experienceby visually presenting virtual objects that are placed in or thatinteract with the real world.

As used herein, VR and AR systems are described and referencedinterchangeably. Unless stated otherwise, the descriptions herein applyequally to all types of MR systems, which (as detailed above) include ARsystems, VR reality systems, and/or any other similar system capable ofdisplaying virtual content (aka “hologram”).

The term “occlusion,” as used herein, refers to scenarios in which oneobject is blocking another object from being viewable inthree-dimensional (3D) space within a MR environment. In the context ofa MR scene, the objects can be real-world objects, holograms, orcombinations of real-world objects and holograms. For instance, ahologram may be structured to occlude a real-world object. Similarly, ahologram may be structured to give the appearance as though a real-worldobject was occluding the hologram. In order to perform realisticocclusion, it is beneficial to closely or even perfectly align theholograms and the scene in order to maintain or provide high levels ofrealism and to provide realistic behavior visualizations.

To provide realistic occlusion effects, MR systems often employ what isreferred to as an “occluder.” An occluder operates by hiding hologramsor holographic content in a manner that would reflect a real-worldscenario. Specifically, an occluder is a type of object that is used toocclude an object's geometry during the rendering and display process.The occluder interacts with the MR scene by providing information to theMR scene's depth buffer. This interaction with the depth buffer enablesthe MR system to prevent certain selected hologram pixels from beingdisplayed in the scene (e.g., pixels that should be hidden from view).FIGS. 1, 2, and 3 will now be referenced to provide someadditional/useful context.

FIG. 1 shows a real-world head 100 and a hologram 105 in the form ofglasses. Notice, the hologram 105 includes a first holographic glassesarm 110 and a second holographic glasses arm 115. In the figure on theleft, the two holographic glasses arms 110 and 115 are currentlydisplayed even though they should be hidden by the real-world head 100(e.g., when the person is “wearing” the holographic glasses, the armswould not be visible from the illustrated perspective). In order toimprove the realism of the hologram 105, MR systems are able to use anoccluder 120 shaped to correspond to the shape of the real-world head100. By enabling the occluder 120 to interact with the MR system's depthbuffer, the MR system can hide the portions of the hologram 105 thatshould be blocked from view by the real-world head 100. The result ofusing the occluder 120 is shown by the hologram 125, where thepreviously displayed holographic glasses arms 110 and 115 are now hiddenfrom view.

FIG. 2 provides another useful illustration of the effects of using anoccluder. Here again is a real-world head 200. In this scenario, anumber of holograms (e.g., hologram 205, hologram 210, hologram 215, andhologram 220) are rotating around the real-world head 200. Although notshown, an occluder is being used, as was the case in FIG. 1 . Based onthe current point of view, the hologram 205 is fully visible and is infront of the real-world head 200. The hologram 210 is also partiallyblocking or occluding view of the real-world head 200. In contrast, thehologram 215 is almost entirely hidden by the occluder, which is shapedto correspond to the real-world head 200. Similarly, a portion of thehologram 220 is also hidden or blocked by the occluder. To clarify,because the holograms are rotating around the real-world head 200 andbecause the holograms 215 and 220 are at positions corresponding to thebackward end of the real-world head 200 (from this perspective), a userwould expect to see various portions of those holograms hidden. Theoccluder provides the ability to hide pixels that should be hidden fromview based on the behavior of the hologram and the content in the sceneor the current perspective.

FIG. 3 illustrates a real-world person 300 wearing an HMD 305.Traditional HMD systems typically utilize on very simplistic occluders,as referenced, for example by the simple occluder 310. Often, the simpleoccluder 310 is formed of basic shapes, such as cylinders, circles,spheres, and so forth. Furthermore, it is often the case that the simpleoccluder 310 fails to account for specific and changing relative posesof the real-world person 300.

In order to generate the simple occluder 310, traditional systemsextensively rely on scanning operations in which an HMD uses its cameras(e.g., perhaps head tracking (HeT) cameras) to scan the environment toidentify real-world objects. By analyzing the camera's images, the MRsystem is able to identify and parse different objects included in theenvironment. Once the MR system has an understanding of the environment,it is able to generate occluders for the different objects. Theoccluders are constantly updated based on constant scanning of theenvironment. Performing this constant scanning is very expensive interms of compute and battery expenditure.

There is currently an ongoing need and desire for improved systems andmethods for generating and using occluders in an MR scene and, even moreparticularly, for improved systems and methods for generating and usingoccluders while also reducing the computational expense that wouldnormally be required to generate and use occluders with conventionalsystems.

The subject matter claimed herein is not limited to embodiments thatsolve any disadvantages or that operate only in environments such asthose described above. Rather, this background is only provided toillustrate one exemplary technology area where some embodimentsdescribed herein may be practiced.

BRIEF SUMMARY

Embodiments disclosed herein relate to systems, devices (e.g., hardwarestorage devices, wearable devices, etc.), and methods for generating,modifying, and utilizing occluders in a MR scene.

Some embodiments dynamically select and present an occluder in a MRenvironment based on assumptions of a user's body in the MR environment.For instance, a relative position of a physical device in the MRenvironment can be identified. This device is one that is being worn orheld by the user, and the relative movements and positioning of thedevice are tracked in the MR environment based on detected sensoroutputs associated with the device. Then, the system determines anassumed understanding of the user's body within the MR environment,including an assumed position of the user's body, based on theidentified relative position of the physical device. Notably, thisunderstanding does not rely on camera imaging of the user's body.

A set of occluders are also provided for selection to be used within theMR environment and to correspond with the assumed position of the user'sbody. Selection is then made of one or more particular occluder(s) fromthe set of occluders based on the assumed understanding of the user'sbody, and a determined relatively higher percentage of correspondence tothe assumed position of the user's body, as compared to other occludersin the set.

Disclosed embodiments also include modifying the presentation of the MRenvironment by positioning the occluder in the MR environment with aparticular size and position corresponding with the assumedunderstanding of the user's body.

Some embodiments also dynamically update an understanding of a user'sbody in a MR environment and modify the occluder(s) for the user basedon updated understanding/assumptions on the assumed position of theuser's body. For instance, some embodiments determine an initialunderstanding of the user's body within the MR environment, where theassumed understanding includes an assumed position of the user's body.An initial occluder is then presented within the MR environment, whichcorresponds with the initial understanding of the user's body and isbased on the assumed position of the user's body. Thereafter, a relativeposition of a tracked device is identified, where the tracked device isworn or carried by the user (e.g., a physical device). Then, based on adetected change in a position or state of use of the tracked device, thesystem updates the understanding of the user's body and correspondingassumption(s) of the user's body position. The detection and updateoperations are based on the detected change in position or state of useof the tracked device and are performed irrespective of any cameraimaging of the user's body. Then, the system modifies the occludershape, position, or selects a new occluder to use within the MRenvironment to reflect the assumed body position of the user based onthe detected change in position/state of the tracked device and thecorresponding determined update(s) about the assumption(s) associatedwith the user's body position.

Some embodiments are also provided to dynamically update a so-called“player collider” for a user within the MR environment based on anunderstanding of the user's body. In such embodiments, for example, theplayer collider includes a defined region associated with the user'sbody. In some instances, an initial understanding of the user's bodywithin the MR environment, including the assumed position of the user'sbody, is determined. This understanding is then used to generate andpresent the player collider within the MR environment (whether visiblyor invisibly). Notably, the player collider corresponds with the initialunderstanding of the user's body and is based on the assumed position ofthe user's body.

Thereafter, an application executing in the MR environment will triggera MR event when an object in the MR environment and the player colliderspatially intersect or collide. In some instances, the system identifiesa relative position of a tracked device that is worn or carried by theuser (e.g., a physical device) and also detects a change in a positionor state of use of the tracked device. The detected relative positionand/or state of the device is then used to update the understanding ofthe user's body and corresponding assumption(s) in the position of theuser's body, irrespective of and without relying on any camera imagingof the user's body. In some instances, the system also modifies a sizeand/or position of the player collider within the MR environment basedon the updated assumptions used to define the understanding of theuser's body position.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

Additional features and advantages will be set forth in the descriptionwhich follows, and in part will be obvious from the description, or maybe learned by the practice of the teachings herein. Features andadvantages of the invention may be realized and obtained by means of theinstruments and combinations particularly pointed out in the appendedclaims. Features of the present invention will become more fullyapparent from the following description and appended claims or may belearned by the practice of the invention as set forth hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which the above-recited and otheradvantages and features can be obtained, a more particular descriptionof the subject matter briefly described above will be rendered byreference to specific embodiments which are illustrated in the appendeddrawings. Understanding that these drawings depict only typicalembodiments and are not therefore to be considered to be limiting inscope, embodiments will be described and explained with additionalspecificity and detail through the use of the accompanying drawings inwhich:

FIG. 1 illustrates an example usage of an occluder to block pixels thatshould be hidden from view.

FIG. 2 illustrates another example usage of an occluder to block pixels.

FIG. 3 illustrates how a simple occluder has traditionally been used toblock pixels.

FIG. 4 illustrates how a set of occluders may be generated and how thoseoccluders may be selected and/or modified in order to provide animproved occluder for use in a MR scene.

FIG. 5 illustrates how an occluder may be modified and/or selected basedon different parameters.

FIG. 6 illustrates some additional parameters that may be used to modifyand/or select an occluder.

FIG. 7 illustrates how an occluder may be generated, modified, orselected based on the detected relative position, orientation, orintended use of an HMD.

FIG. 8 illustrates an example scenario in which an HMD is being used inconjunction with another tracked object (e.g., a handheld device).

FIG. 9 illustrates how an occluder can be generated, modified, orselected based on a combination of pose information obtained for boththe HMD and the handheld device/instrument, where the resulting occluderis configured based on an assumed pose that is determined from positioninformation associated with the handheld instrument.

FIG. 10 illustrates how an occluder may be generated or assembled fromany number of sub-portions or sub-occluders.

FIG. 11 illustrates how detected movement of a handheld instrument cantrigger modification to the occluder.

FIG. 12 illustrates how the state of a handheld instrument can triggermodification to the occluder.

FIG. 13 illustrates how occluders can be shared amongst other HMDs in amulti-user scenario.

FIG. 14 illustrates how each HMD is able to maintain informationregarding the occluders of other persons or objects in the MR scene.

FIG. 15 illustrates an example MR scene in which a hologram is beingdisplayed and in which the hologram is being occluded by differentoccluders present (though not visible) in the scene.

FIG. 16 illustrates an example use case in which a player collider (i.e.a type of occluder) is present in the scene in order to determinewhether a hologram (e.g., the dragon's fire breath) is striking thereal-world person.

FIG. 17 illustrates a flowchart of an example method for dynamicallyselecting and presenting an occluder in a MR environment/scene, wherethe occluder is configured based on assumptions related to the user'sbody.

FIG. 18 illustrates a flowchart of an example method for dynamicallyupdating an understanding of the user's body and for modifying anoccluder associated with that user's body, where the occluder isconfigured based on an assumed position of the user's body.

FIG. 19 illustrates a flowchart of an example method for dynamicallyupdating a so-called “player collider” or “hit box” of a user, where theplayer collider is configured based on an understanding of the user'sbody.

FIG. 20 illustrates an example computer system that is configurable toperform any of the disclosed operations.

DETAILED DESCRIPTION

Embodiments disclosed herein relate to systems, devices (e.g., hardwarestorage devices, wearable devices, etc.), and methods for configuringoccluders in a MR scene.

In some instances, the disclosed systems and methods are capable ofbeing used to generate and use occluders with reduced computationalexpense relative to conventional systems and methods for generating andusing occluders based on continuous scanning & imaging of objectsassociated with the occluders.

Some embodiments dynamically select and present an occluder in a MRenvironment based on assumptions of a user's body. For instance, arelative position of a physical device can be identified. The relativemovements and positioning of the device are tracked based on sensoroutputs. An assumed understanding of the user's body is determined basedon the relative position of the device, thereby enabling an assumptionto be made regarding the position of the user's body. A set of occludersare identified and are made available for presentation within the MRenvironment. These occluders are configured to correspond with theassumed position of the user's body. The embodiments select a particularoccluder and then modify the presentation of the MR environment bypositioning the occluder in the MR environment to have a size andposition corresponding to the assumed position of the user's body.

Some embodiments dynamically update an understanding of a user's bodyand modify an occluder for the user based on an assumed position of theuser's body. For instance, an initial understanding of the user's bodyposition is determined. An occluder based on the assumption is presentedwithin the MR environment. A relative position of a tracked device isidentified. The embodiments detect a change in a position or state ofthe tracked device and then update the understanding of the user's bodyposition. The embodiments also modify the occluder within the MRenvironment based on the update.

Some embodiments dynamically update a “player collider” based on anunderstanding of the user's body. For instance, an initial understandingof the user's body position is determined. The player collider ispresented within the MR environment. The player collider correspondswith the initial understanding of the user's body and is based on theassumed position of the user's body. An application executing the MRenvironment (or perhaps in the MR environment) will trigger a MR eventwhen an object in the MR environment and the player collider spatiallyintersect or collide. A relative position of a tracked device isidentified. The embodiments update the understanding of the user's bodyand corresponding assumption based on the position or state of thetracked device. The embodiments also modify a size or position of theplayer collider based on the update.

Examples Of Technical Benefits, Improvements, And Practical Applications

The following section outlines some example improvements and practicalapplications provided by the disclosed embodiments. It will beappreciated, however, that these are just examples only and that theembodiments are not limited to only these improvements.

The disclosed embodiments bring about substantial improvements andbenefits to the technical field and also provide numerous practicalapplications. For instance, the embodiments are able to dynamicallyconfigure an occluder based on position information obtained withouthaving to perform iterative and costly (in terms of compute and batteryusage) scanning operations for generating and/or updating theunderstanding of the user's body position and corresponding occluder(s)to be used.

Instead, the disclosed embodiments can be used to determine the relativepose (e.g., position and orientation) of an HMD and use that poseinformation to make an intelligent assumption regarding thecurrent/updated pose of the person. For instance, if the HMD has aparticular pose, then the embodiments are able to determine how theperson is likely orientated in order to enable the HMD to have thatspecific pose. If the person is holding or wearing another trackabledevice, then the pose of that trackable device can also be factored intothe assumption calculus. Once the person's body position is assumed, anoccluder can be configured to have a size, shape, and orientationcorresponding to the person's assumed body position. As a consequence,the disclosed embodiments are able to refrain from performing continuousscans of the environment in order to determine how to structure orconfigure an occluder.

Even if an initial set of one or more scans/images is used to develop aninitial understanding of the user's body position, the currentembodiments can be used to make assumptions for updating anunderstanding of the user's body position without relying on newimages/scans of the user's body and by relying, instead, on assumptionsbased on detected positioning of tracked devices being held/used by theuser in the MR environment. By refraining from taking and/or usingadditional scans of the user's body, while updating the understanding ofthe user's body position, it is possible to reduce the computationalexpense associated with generating and using occluders in MRenvironments relative to conventional systems that rely on imaging ofthe user's body to update the understanding of the user's body position.

Accordingly, the disclosed embodiments significantly improve thetechnical field by providing techniques that reduce battery consumptionand reduce compute usage. The embodiments also improve the technicalfield by providing the ability to dynamically and in real-time modifyoccluders based on changing conditions in the MR scene (e.g., changes inthe user's body pose) with reduced computational expense relative toconventional systems.

Configuring Occluders Based on Assumed Body Positions

Instead of scanning an environment in order to determine an object'spose (i.e. location and orientation) based on camera image data, thedisclosed embodiments are able to utilize pose information obtained fromnon-camera sensors in order to determine an object's or person's pose.Examples of non-camera sensors include, but are not limited to, inertialmeasurement units (IMUS) (including accelerometers, gyroscopes,magnetometers, etc.), global positioning systems (GPS), and others.

For instance, an HMD may be configured to have the non-camera sensorsand can use those sensors to determine the HMD's pose. Based on thatpose information, the embodiments are able to assume or estimate thepose of the person who is wearing the HMD without scanning the person.Once that assumption is performed, the embodiments are then able toconfigure an occluder to have a shape and pose corresponding to theperson's assumed pose.

In some cases, the embodiments generate and configure an occluder whilein other cases the embodiments first select an initial pre-generatedoccluder and then modify or further configure that occluder based on theassumed pose. FIG. 4 is illustrative of some of these processes.

Some embodiments maintain a library 400 comprising a set of occluders405, such as occluder 410, occluder 415, and occluder 420. Of course,any number of occluders may be included in the set of occluders 405. Theset of occluders 405 may be configured to correspond to any number ofdifferent human body poses. For instance, one occluder in the set ofoccluders 405 may be configured to have a pose corresponding to that ofa standing male, based on average human male characteristics (e.g.,height, weight, shoulder width, torso width, etc.). Another occluder maybe configured to have a pose corresponding to that of a standing female,based on average human female characteristics. Another occluder may beconfigured to have a pose corresponding to a crouching individual orperhaps even a person laying on the ground. Indeed, any number ofdifferent poses realized by an occluder. That is, similar to how theoccluder 120 from FIG. 1 generally matched or corresponded to the shapeand dimensions of the real-world head 100, the set of occluders 405 mayhave any number of different occluders configured to correspond to anynumber of shapes and poses.

In some embodiments, the sophistication, complexity, or level ofgranularity of an occluder may be high and may closely approximate thepose, posture, and features of a person. In some embodiments, thecomplexity or level of granularity may be lower such that the shape ofthe occluder abstractly mimics the shape of a person. For instance, abasic occluder may have a basic circle or sphere for a person's head anda basic cylinder or rectangle for the person's torso. The arms and legsmay also be formed using basic shapes. In more granular or more complexoccluders, the arms, legs, head, and torso of the occluder may bedesigned to closely approximate a person's actual arms, legs, head, andtorso. It may be the case that an initial calibration operation isperformed where the HMD performs a scan of the person's body in anattempt to identify the physical features of the person. Based on thatinitial calibration, some embodiments then design a highly detailed andintricate occluder designed to mimic the detected features of theperson.

FIG. 4 illustrates a simplified flowchart of some of the disclosedprinciples. FIG. 4 introduces these concepts at a high level while theremaining figures provide further details and clarification.

The disclosed embodiments initially identify a set of parameters 425,such as the pose of the HMD being worn by a person. A detaileddiscussion on parameters will be provided later. Based on the parameters425, the embodiments then perform an estimation in order to determine orassume the person's body pose, as shown by assumed body position 430.Based on the person's assumed body position, the embodiments thenconfigure an occluder to have a shape and configuration matching theassumed body pose.

In some instances, the embodiments generate from scratch an entirely newoccluder while in other instances the embodiments select a pre-generatedoccluder and then potentially modify that occluder to match the assumedbody pose. The box labeled modifications 435 is symbolic of either oneor both of those two operations. As a result of performing themodifications 435, the embodiments produce a modified occluder 440having a specific configuration (e.g., shape, orientation, and posematching that of the assumed body position). FIGS. 5 and 6 provide someadditional details regarding some of the parameters 425.

FIG. 5 shows parameters 500, which are representative of the parameters425 from FIG. 4 . In particular, FIG. 5 shows an HMD 505 that iscurrently positioned at a height 510. Similarly, FIG. 5 shows adifferent HMD 515 that is currently positioned at a height 520.

Using sensor(s) 525 included in the HMDs 505 and 515, the embodimentsare able to not only determine the heights 510 and 520 but otherpose-related information as well. In some cases, the sensor(s) 525include, but are not limited to, an IMU 530 and a head tracking HeTcamera 535. The ellipsis 540 demonstrates how other sensors (e.g., GPS,etc.) may be used as well. While this disclosure previously stated thatnon-camera sensors may be used, it should be noted that the disclosedembodiments are not prevented from using camera sensors as well, such asthe HeT camera 535. For instance, the disclosed embodiments are able toperiodically rely on camera image data in order to ensure that thenon-camera sensors are still in sync with the camera imagery.Accordingly, camera images may be used to supplement, augment, orperhaps synchronize the data from the non-camera sensors. Furtherdetails of this aspect will be provided later.

In any event, the embodiments are able to determine the HMD 505 and515's 6 degree of freedom (6 DOF) pose 545, which includes translationinformation (e.g., back/forward, left/right, and up/down) and angularinformation (e.g., yaw, pitch, and roll). Based on the 6 DOF pose 545,an estimation or assumption can be made regarding the person's own bodyposition or pose. For instance, in order to have the HMD 505 be at theillustrated location, the embodiments assume the woman is standing.Similarly, in order to have the HMD 515 be at the illustrated location,the embodiments assume the man is crouching.

More specifically, in FIG. 5 , the HMD 505 on the woman appears to betilted slightly upward while the woman is standing. Because the HMD 505has determined the height 510 and because the HMD 505 has determined theorientation or angular pose of itself, the HMD 505 is able to assumethat the woman is in a standing pose with her head slightly tiltedupward.

Similarly, the HMD 515 on the man appears to be faced straightforwardwhile the man is crouching. Because the HMD 515 has determined theheight 520 and because the HMD 515 has determined the orientation orangular pose of itself, the HMD 515 is able to assume that the man is ina crouching pose with his head relatively straight.

FIG. 6 lists some additional parameters 600 that may be included amongthe parameters 425 of FIG. 4 , and which parameters are used to assume aperson's body position. Specifically, parameters 600 include an intendeduse 605 parameter, an activity/actual use 610 parameter, a relativemovement(s) 615 parameter, and a relative positioning 620 parameter. Theellipsis 625 illustrates how other parameters may be considered as well.

The intended use 605 parameter generally refers to an activity that theperson appears to be engaged in or is planning on being engaged in. Forinstance, if the HMD detects that the person is holding a golf club,then the HMD can estimate or predict that the person is about to play around of golf. Based on that assumption, the HMD can proceed withconfiguring a number of occluders that are generally shaped tocorrespond to a person's pose while playing golf. One example pose wouldbe the backswing of the person. The embodiments are able to configure anoccluder to have a pose corresponding to the backswing pose. Of course,that is only one example, and the broader principles should be applied.In this regard, the embodiments are able to detect, determine, orestimate an intended use 605 regarding how the HMD will be used and thenconfigure occluders based on that intended use 605.

The activity/actual use 610 parameter refers to an activity that iscurrently being performed by the person wearing the HMD. Any activitymay be detected, such as perhaps cooking a dinner or sitting on a couchor lying in a bed. Based on the detected activity, the embodiments arealso able to configure occluders.

The relative movement(s) 615 refers to detected movements of the HMD andperson. Perhaps a specific activity is not currently known by the HMD.Despite the possibility of not detecting which specific activity theperson and HMD are currently engaged in, the embodiments are still ableto track the relative movement(s) 615 of the HMD and person in order toestimate or assume the person's body position. Similarly, theembodiments are able to detect the relative positioning 620 of theperson in order to configure occluders.

FIG. 7 illustrates various different examples of actual occluders andhow those occluders may be configured based on a person's assumed bodyposition. FIG. 7 shows a device 700, which is representative of any ofthe HMDs discussed thus far. Based on the detected or determinedparameters of the device 700, the embodiments are able to generate anassumed understanding 705 regarding the person's body pose or position.Based on the assumed understanding 705, the embodiments are able toconfigure an occluder to have characteristics matching the assumed poseof the person wearing the device 700, as shown by the occluder 710.

Notice, the occluder 710 has a shape, form, and orientation generallymatching the shape, form, and orientation of the person wearing thedevice 700. Furthermore, this occluder 710 was generated without relyingon camera imagery obtained from a scan; rather, the occluder 710 wasconfigured based on an assumed body position of the person, where theassumption is based on the detected pose (i.e. location and orientation)of the device 700. In the context of device 700, the assumedunderstanding 705 of the user's body comprises (in this case) adetermination that the user is standing. That determination may furtherinclude a determination regarding how tall the user is, what posture orpose the user has, and so forth.

Because the embodiments do not necessarily rely on camera imagery, theremay be a few slight discrepancies between the pose of the occluder 710and the pose of the person. For instance, notice the woman's arm 715 isslightly forward relative to her body. The occluder 710, on the otherhand, has the arm 720 near the torso such that there is a slightdiscrepancy between the pose of the occluder 710 and the pose of thewoman. The disclosed embodiments are enabled to address this issue in anumber of ways, including (i) periodically using camera imagery toensure alignment and/or (ii) relying on pose information from a trackedobject that is either being held or worn/carried by the person. Theseaspects will be discussed in more detail later. Additional operationsmay also be used/performed to further mitigate and/or reduce thepossibility of discrepancies between the pose of the occluder 710 andthe pose of the actual person wearing the device 700.

FIG. 7 shows another device 725 being worn by a person. Similar to thediscussion regarding device 700, the embodiments are able to generate anassumed understanding 730 of the person's pose based on parametersdetected with regard to the pose of the device 725. Based on the assumedunderstanding 730, the embodiments can then generate an occluder 735having a pose that is assumed to match or coincide with the assumed poseof the person wearing the device 725.

In the context of device 725, the assumed understanding 730 of theuser's body comprises a determination that the user is crouching (or, inother scenarios, perhaps laying down or any other position). In somecases, that determination may further include a determination of arelative direction the user is crouching or laying within the MRenvironment.

Accordingly, the embodiments are able to obtain sensor data detailingthe pose of an HMD and/or another device. Using that sensor data, theembodiments determine the HMD's pose and then make an assumptionregarding how the person wearing the HMD is likely positioned in orderto have the HMD be in its pose. After estimating or assuming theperson's pose, the embodiments then configure an occluder to havecharacteristics or features matching that assumed body pose. Byperforming these operations, the embodiments are able to refrain fromconstantly having to scan the environment to generate occluders.

Configuring Occluders Based on Pose Information from Multiple Devices

In an effort to avoid the scenario shown in FIG. 7 where the arm 720portion of the occluder 710 does not mimic the arm 715 of the personwearing the device 700, some embodiments are configured to track andmonitor a second device that is associated with the person wearing theHMD. Such a scenario is shown in FIG. 8 .

FIG. 8 illustrates an example environment 800 in which an HMD 805 isoperating. HMD 805 is representative of the HMDs discussed thus far inthat the HMD 805 is able to acquire sensor information in order todetermine a pose of the HMD 805 and then use that pose to predict,estimate, or assume a pose of the person wearing the HMD 805. Asdiscussed previously, it is beneficial to track an additional device onthe person in order to improve the assumption regarding the person'sbody position. As such, FIG. 8 shows an example scenario in which theperson wearing the HMD 805 is also holding (or perhaps wearing) atracked object 810.

Tracked object 810 may be any type of object or device that is capableof being tracked by the HMD 805 and/or capable of performing its owntracking. For instance, suppose the tracked object 810 is anon-electronic device. In some cases, therefore, the HMD 805 may use itsHeT cameras or other cameras to record the position of the trackedobject 810 in order to improve the resulting body position assumption.

In other scenarios, the tracked object 810 is a smart device capable ofusing its own sensor data to determine its pose. For instance, similarto how the HMD 805 may include IMUs, a GPS, and other sensors, thetracked object 810 may also include similar types of sensors. By way ofexample and not limitation, the tracked object 810 may be a smart phone,tablet, laptop, Internet-of-Things (IoT) device, or some other so-called“smart” device.

The HMD 805 and the tracked object 810 are able to communicate over awireless connection (e.g., NFC, Bluetooth, wideband radio connection,etc.) or perhaps even a wired connection. The tracked object 810 is ableto use this connection to transmit its pose information to the HMD 805.Based on the received pose information from the tracked object 810, theHMD 805 is able to then identify where the tracked object 810 is locatedin relation to the HMD 805. Based on that relative location and how thetracked object 810 is oriented, the HMD 805 is able to generate anassumption regarding how the person wearing the HMD 805 is likelypositioned in order to cause the tracked object 810 to have thedetermined orientation and placement location and in order to cause theHMD 805 to have the determined orientation and placement location. FIG.9 is illustrative of these concepts.

FIG. 9 shows an HMD 900, which is representative of the HMD 805 fromFIG. 8 , and a handheld instrument 905, which is representative of thetracked object 810 from FIG. 8 . As shown, the person wearing the HMD900 has his arm 910 angled in order to hold the handheld instrument 905at the indicated orientation and location (i.e. pose).

In accordance with the disclosed principles, the handheld instrument 905is able to determine its 6 DOF pose 915 and then transmit this poseinformation to the HMD 900. The HMD 900 is also able to determine its 6DOF pose. The combination of these two pieces of information enables theHMD 900 to determine the relative position of the handheld instrument905 relative to the HMD 900. Based on that relative position, the HMD900 is then able to generate an assumption or an estimate regarding howthe person is likely to be positioned (e.g., body pose assumption 920)to enable both the HMD 900 to be at its specific pose and the handheldinstrument 905 to be at its specific pose. Based on the assumed bodyposition, the embodiments are then able to generate a correspondingoccluder that is configured to match or coincide with the positionalarrangement of the person, or rather the person's assumed body position.

FIG. 9 shows the resulting occluder in the form of occluder 925. Notice,the shape, size, and orientation of the occluder 925 is configured in aneffort to match the shape, size, and orientation of the person wearingthe HMD 900, based on the body pose assumption 920. Furthermore, becausethe HMD 900 was able to utilize the pose information from the handheldinstrument 905, the embodiments were able to improve or further refinethe occluder 925 to have a higher degree of correlation, specificallywith regard to the arm 910. Here, the assumed arm position occluder 930is configured in a manner to correspond to the position and angling ofthe arm 910. This modification to the occluder 925 is performed as aresult of obtaining the 6 DOF pose 915 from the handheld instrument 905and as a result of generating an assumption regarding how the arm 910 islikely to be positioned in order to achieve the 6 DOF pose 915.

In some embodiments, the assumed understanding of the user's body (e.g.,the body pose assumption 920) is based on the relative position of afirst physical device (e.g., the HMD 900 or the handheld instrument 905)in a MR environment in combination with a determined relative positionof a second physical device (e.g., the other one of the HMD 900 or thehandheld instrument 905) in the MR environment.

In some embodiments, the resulting occluder (e.g., occluder 925) may begenerated from multiple constituent subparts or sub-occluders that aregrouped or fused together to form the overall occluder. Consequently,some embodiments perform an occluder assembly 1000, as shown in FIG. 10.

FIG. 10 shows the occluder 1005, which is representative of the occluder925 from FIG. 9 . In some embodiments, an initial base occluder isgenerated based on the pose information of the HMD. For instance, if thepose information of the HMD indicates the person wearing the HMD islikely standing, then an initial standing pose occluder may be selected.That initial occluder may then be subsequently modified based on thepose information of the handheld instrument.

For instance, the library 400 of FIG. 4 may include any number ofoccluder parts that may be mixed and matched with other occluder partsin order to form the final occluder. By way of example, the occluderpart 1010, occluder part 1015, occluder part 1020, and occluder part1025 may all be stored in the library 400.

In some implementations, the embodiments analyze the available occluderparts and select one that will result in the highest degree or level ofcorrespondence 1030 in order to match the pose of the handheldinstrument. For instance, the occluder part 1025 is shown as being anarm in the downward position, so it is unlikely that the person's armwould be in that pose because that pose would not enable the handheldinstrument to assume the 6 DOF pose illustrated in FIG. 9 . Similarly,the occluder parts 1020 and 1015 are also unlikely to enable thehandheld instrument to assume the illustrated pose.

The occluder part 1010, on the other hand, would provide the highestcorrespondence 1030 or confidence in achieving the pose of the handheldinstrument in FIG. 9 . Consequently, the embodiments are able to selectthe occluder part 1010 and affix or fuse it to any other occluder partsin order to form the resulting occluder 1005. In some cases, theselected occluder part may still not perfectly or sufficiently alignwith the actual pose of the handheld instrument (e.g., a thresholdrequirement may not be met). In such cases, the embodiments are able tomodify the selected occluder part in an effort to better align it withthe assumed body position.

While FIG. 10 illustrated a scenario involving an “arm” occluder part,one will appreciate how an occluder part may correspond to any bodypart. For instance, there may be occluder parts for heads, arms, torsos,legs, feet, and so forth, without limit. These different body partoccluders may be for different genders, weight specifications, heightspecifications, orientations, poses, and so forth.

In some cases, instead of selecting pre-generated occluders and thenmodifying and/or fusing them, some embodiments build or generateoccluders from scratch based on the pose information of the HMD and thehandheld instrument. As such, the principles described herein should beinterpreted broadly.

It is typically the case that people in a MR scene are not stationary.As such, the embodiments are able to dynamically and in real-time adjustthe occluders based on the detected movements of the person. FIG. 11 isillustrative.

FIG. 11 shows an example scenario in which a handheld instrument 1100,which is representative of the handheld instrument 905 of FIG. 9 , isbeing held by a person wearing an HMD. Based on this current pose, theembodiments have generated or configured a corresponding occluder 1105.Subsequently, there is a detected movement 1110 of the handheldinstrument such that it is now at the position shown by handheldinstrument 1115. The embodiments are able to collect pose informationfrom the handheld instrument 1100 in order to determine when, how, andwhere the handheld instrument 1100 is moving. Based on that poseinformation, the embodiments can then generate an assumption regardingthe new position of the person holding the handheld instrument. Newassumptions and new occluders may be generated even while the movementis occurring.

In response, the embodiments can then generate a new occluder 1120 basedon the updated pose. Notice, the arm holding the handheld instrument inthe occluder 1120 is now in a relaxed position to correspond to the newposition of the handheld instrument 1115. Accordingly, the embodimentsare able to dynamically modify or configure occluders based not only ondetected movements of an HMD but also based on detected movements of ahandheld instrument.

Some embodiments periodically or on a triggered basis perform an update1125 to the understanding or assumption of the person's body positionusing the HMD's cameras in order to obtain camera-based sensor data1130. That is, some embodiments operate in an effort to maintainsynchronization between the person's actual pose and the poserepresented by an occluder by periodically using camera imagery toensure alignment.

In some implementations, the occluder may be configured based on acurrent state of the handheld instrument. FIG. 12 is illustrative ofthis concept. FIG. 12 shows a handheld instrument 1200, which isrepresentative of the handheld instruments discussed thus far. In somecases, the embodiments are able to determine a state 1205 of thehandheld instrument 1200 and then configure an occluder based on thestate 1205, as shown by the occluder 1210.

By way of example, suppose the handheld instrument 1200 was an arrow andthe person wearing the HMD was also wearing a quiver. If the state 1205of the arrow was that of the arrow being out of the quiver, then theembodiments can infer or predict an intended use of the arrow andconfigure the occluder 1210 to have a shape and configuration based onthe likelihood that the arrow will be strung on a bow string and used inthat manner. On the other hand, if the state 1205 of the arrow was thatof the arrow being inside of the quiver, then the embodiments can inferthat the arrow is likely not to be used and can then refrain fromcontinuing to collect pose information for that arrow, at least for aperiod of time. If the state 1205 changes, then the data collectionprocess may resume.

As another example, suppose the handheld instrument 1200 was a golfclub. If the state 1205 of the golf club was that of the golf club beingheld in the person's hands, then the embodiments can infer that the golfclub is about to be used in accordance with conventional golf club usagetechniques. The embodiments can then configure the occluder 1210 basedon the state 1205 of the golf club. Relatedly, if the golf club, whichis being tracked, is not being used in accordance with club usagetechniques (e.g., perhaps it is placed in a golf bag), then theembodiments can identify that state of the club and configure theoccluder 1210 accordingly.

Yet another example could be that of an active or inactive state of thehandheld instrument 1200. For instance, the active state may be apowered-on state or perhaps a demonstrated usage of the handheldinstrument 1200. On the other hand, the inactive state may be a poweredoff state or a non-use of the handheld instrument 1200. Any detectablestate may be relied on in order to configure the occluder 1210.

Accordingly, in some embodiments, the assumed understanding of theuser's body is based on the relative position of a physical device(e.g., an HMD or handheld device) in combination with a determined stateof intended use or actual use of the physical device in the MRenvironment. Of course, multiple parameters or factors may beconcurrently considered when generating the assumption of the person'sbody position. In some cases, the assumed understanding of the user'sbody within the MR environment includes determining an assumed positionof an arm, leg, torso, or other body part of the user based on thedetermined state of the handheld instrument within the MR environment.In some cases, the embodiments detect a change in the determined stateof the handheld instrument within the MR environment and then modify atleast one of a selection of an occluder or a shape or position of theoccluder based on the detected change in the determined state of thehandheld instrument. Such operations may be performed irrespective ofany change in relative positioning of the handheld instrument within theMR environment.

Sharing Occluders with Other HMDs

Often, HMDs are used in a multi-user scenario. It is beneficial to beable to share an occluder generated by one HMD with any other HMDslocated in the same environment.

FIG. 13 shows an environment 1300 in which multiple HMDs are present, asshown by HMD 1305, HMD 1310, and HMD 1315. Each of these HMDs may beconfigured in the manner described earlier in that each is able togenerate its own respective occluder based on pose information of theHMD and/or pose information of a handheld instrument or other trackedinstrument on the person's body. As recited above, it is oftenbeneficial to share this occluder information with other HMDs so theycan benefit by using that occluder instead of performing a scan of theroom.

As an initial matter, in some cases, the HMDs may individually orcollectively perform an initial scan of the environment 1300 todetermine properties of the environment 1300 and to generate a spatialmapping or awareness of the environment 1300. Once that initial scan isperformed, the embodiments can refrain from performing a continuous scanand instead can rely on the principles taught herein to generateoccluders and to place holograms in the MR scene. If the scanning hasstopped, then one HMD might not know the current position of anotherHMD. As such, it is beneficial to share the occluders between thedifferent HMDs in order to enable each HMD to have data detailing thepose of the person wearing each HMD and in order to enable each HMD toplace holograms at realistic locations using the occluders. Furtherdetails on this aspect will be provided later.

In FIG. 13 , the HMD 1305 is communicating with the HMD 1310 and thosetwo HMDs are passing back and forth occluder(s) 1320. To clarify,occluders for the person wearing the HMD 1305 are being transmitted tothe HMD 1310, and occluders for the person wearing the HMD 1310 arebeing transmitted to the HMD 1305. Similarly, HMD 1305 and HMD 1315 aretransmitting occluder(s) 1325 back and forth, such that HMD 1305 hasoccluder information for the person wearing HMD 1315 and HMD 1315 hasoccluder information for the person wearing HMD 1305. To complete theexample, HMD 1315 and HMD 1310 are transmitting occluder(s) 1330 backand forth, such that HMD 1310 has occluder information for the personwearing HMD 1315 and HMD 1315 has occluder information for the personwearing HMD 1310. Any type of wireless connection may be used tofacilitate these communications.

In this regard, the different HMDs are able to communicate with oneanother to deliver or transmit occluder information to one another. Indoing so, the HMDs are able to have live and up to date occluders forthe various different HMDs in the environment 1300, and each HMD is ableto use those occluders to interact with their respective depth buffersin order to place holograms in the MR scene.

FIG. 14 shows an environment 1400, which is representative of theenvironment 1300 of FIG. 13 . Now, each HMD has occluder information forthe other HMDs in the environment 1400. For instance, each HMD is awareor has information reflective of occluder 1405, occluder 1410, andoccluder 1415. The size 1420, position 1425, and shape 1430 of eachoccluder are also known by each HMD. Furthermore, as the occluders areupdated or further configured based on movements of the HMD and othertrackable objects, each HMD receives the updated information in order tomaintain an up to date occluder for each respective HMD.

Occluders Influence in MR Scenes

Occluders are beneficial because they enable or provide for the improvedplacement of holograms in a MR scene/environment. By knowing thepositional and depth information of objects in the scene, where eachobject has its own respective occluder, the embodiments are able toprovide realistic holograms or holographic visualizations reflective ofreal-world occurrences. FIG. 15 is illustrative of this concept.

FIG. 15 shows a MR environment 1500 in which the different HMDs of FIG.13 are operating, including HMD 1505 (representative of HMD 1305). Thedifferent HMDs have and/or are transmitting their respective occludersto the other HMDs such that each HMD can utilize the occluders forinteraction with their respective depth buffers. In this regard, the HMD1505 is aware of the person 1510 and the occluder corresponding to thatperson (e.g., occluder 1410 from FIG. 14 ) and the person 1515 and theoccluder corresponding to that person (e.g., occluder 1415).

Using that information, the HMD 1505 is generating a hologram 1520 inthe form of a dragon. Notably, the hologram 1520 is generated from thepoint of view of the HMD 1505. Notice also, the hologram 1520 isgenerated to have the appearance of being physically between the person1515 and the person 1510.

That is, from the point of view of the HMD 1505, the hologram 1520 ismore distant relative to the person 1510 but is less distant relative tothe person 1515. Here, the person 1510, or rather the occluder 1410, iscausing some pixels of the hologram 1520 to be hidden or occluded (e.g.,the leg of the dragon is partially hidden) while the hologram 1520 isitself occluding the person 1515. The occluder 1410, which correspondsto the person 1510, is providing the ability to realistically hide orprevent certain pixels of the hologram 1520 from being visuallydisplayed, thereby giving the illusion or impression that the dragon isphysically farther away from the HMD 1505 as compared to the person1510.

In addition to using occluders to improve how holograms are visuallydisplayed, the embodiments are also able to use occluders to determinewhen a hologram virtually “contacts” a person via use of a so-called“player collider” or “hit box.” FIG. 16 is illustrative.

FIG. 16 shows a MR environment 1600, which is representative of the MRenvironment 1500 of FIG. 15 . Using the occluder 1405 from FIG. 14 , theembodiments are able to generate the player collider 1605, which is usedto determine whether or not virtual content has touched or struck aperson.

For instance, the holographic dragon is shown in FIG. 16 as emitting aholographic flame 1610. Using the player collider 1605, the embodimentsare able to determine or discern where, when, and how the holographicflame 1610 virtually contacts the person wearing the HMD. In thisscenario, the holographic flame 1610 is contacting the person as shownby the strike area 1615 and the strike area 1620.

In some embodiments, the player collider 1605 may be shaped, oriented,or configured to correspond to the shape of the person's respectiveoccluder. In this case, the occluder can operate as the player collider.

In some embodiments, the player collider 1605 may be abstracted orsimplified to achieve various levels of granularity. For instance, aplayer collider mimicking the shape of a person's actual pose will havea high level of granularity while a player collider formed in the shapeof a box (as shown in FIG. 16 ) will have a lower level of granularity.In this regard, the player collider can be thought of as a less detailedor simplified occluder.

The player collider 1605 is beneficial because it can be used by the HMDto determine when holograms virtually contact a person wearing the HMD.For instance, the HMD will be able to discern that the dragon's firebreath is striking the person at multiple locations. If this MR scene isa part of a game, then the HMD can use the strike areas 1615 and 1620 toperhaps lower the gaming life of the player playing the game (e.g.,because he was hit with fire). Of course, the concepts can be applied inother scenarios as well, such as first responder training and so forth.

The action of the holographic flame 1610 striking the person constitutesa MR event 1625 and may be used to impact subsequent play of the game orsubsequent visual effects in the MR scene. That is, triggering the MRevent 1625 may alter how the MR scene behaves when holographic contentvirtually touches or strikes a person wearing the HMD. One example couldbe to visually display a red hologram when the user is struck with fire.Of course, these are examples only, and the broader principles should beapplied.

EXAMPLE METHOD(S)

The following discussion now refers to a number of methods and methodacts that may be performed. Although the method acts may be discussed ina certain order or illustrated in a flow chart as occurring in aparticular order, no particular ordering is required unless specificallystated, or required because an act is dependent on another act beingcompleted prior to the act being performed.

Attention will now be directed to FIG. 17 , which illustrates aflowchart of an example method 1700 for dynamically selecting andpresenting an occluder in a mixed-reality environment, where theoccluder is designed or configured based on assumptions of a user's bodyin the mixed-reality environment. Initially, method 1700 includes an act(act 1705) of identifying a relative position of a physical device in aMR environment. Here, the physical device is being worn or held by auser. An example of the physical device may be any of the HMDs and/orhandheld instruments mentioned earlier. Additionally, the relativemovements and positioning of the physical device are being tracked inthe MR environment based on one or more sensor outputs.

For instance, suppose the physical device is the HMD 505 shown in FIG. 5(i.e. the physical device comprises an HMD device that is worn by theuser). The relative movements and positioning of the HMD 505 are beingtracked using the sensor(s) 525 in order to determine the 6 DOF pose 545of the HMD 505. In some cases, the physical device comprises a handheldinstrument, as was discussed earlier. In some cases, the physical devicemay actually be multiple devices operating in conjunction with oneanother, such as the HMD and the handheld device.

Method 1700 continues with an act (act 1710) of determining an assumedunderstanding of the user's body within the MR environment. This assumedunderstanding is based at least in part on the identified relativeposition of the physical device in the MR environment and is assumedwithout a reliance on camera imaging (or at least constant cameraimaging) of the user's body. Furthermore, the assumed understanding ofthe user's body includes at least an assumed position of the user'sbody. Using FIG. 7 as an example, the person wearing the device 725 isillustrated as being in a crouched position. Based on the sensor datadetailing the 6 DOF pose of the device 725 and perhaps based on dataacquired from an initial calibration (e.g., to determine whether theperson is a male, female, adult, child, etc.), the embodiments are ableto generate an assumed understanding 730 of the person's pose, whichassumed understanding 730 is based at least in part on the pose of thedevice 725.

Method 1700 then includes an act (act 1715) of identifying a set ofoccluders that are available for selection for being presented withinthe MR environment to correspond with the assumed position of the user'sbody. As described earlier, the embodiments are able to maintain alibrary 400 of occluders that may initially be used and perhaps laterrefined, configured, or modified in an effort to more fully align theselected occluder with the assumed body position. By “present,” it ismeant that the occluder is permitted to operate with the MRenvironment's depth buffer. In some cases, “present” can includerefraining from visually displaying the occluder while in other cases“present” can include visually displaying the occluder or a portion ofthe occluder.

Method act 1720 involves selecting a particular occluder from the set ofoccluders. The selection of the particular occluder is based on theassumed understanding of the user's body within the MR environment. Theselection may also be performed by selecting the particular occluderthat is determined to have a higher percentage of correspondence to theassumed position of the user's body as compared to other occluders inthe set of occluders. For instance, the occluder part 1010 in FIG. 10was determined to have a higher percentage of correspondence 1030relative to the person's arm as compared to the occluder part 1025.

In some implementations, the occluder may comprise a player colliderbox. In this scenario, the process of presenting the occluder in the MRenvironment may potentially involve defining a relative position of theplayer collider box within the MR environment without generating orrendering a hologram within the MR reality environment to reflectplacement of the player collider box. That is, the player collider boxmay be transparent in the MR scene, yet it is used to determine when ahologram has virtually touched or contacted the user.

Method 1700 then includes an act (act 1725) of modifying thepresentation of the MR environment by positioning the occluder in the MRenvironment with a particular size and position corresponding with theassumed understanding of the user's body. It should be noted that“positioning” the occluder in the MR environment does not necessarilymean that the MR environment will visually display the selectedoccluder. Instead, in this context, “positioning” the occluder refers tothe case where the occluder interacts with the system's depth buffer sothat the MR environment will accurately reflect the placement ofholograms. Although in some cases the occluder may be visually displayed(e.g., the process of modifying the presentation of the MR environmentmay include rendering the occluder as a hologram), it is typically thecase that the occluder is fully transparent or not visible to the personimmersed in the MR scene. That is, the occluder operates in thebackground to ensure that the visual display of holograms accuratelyreflects real-world expectations (in terms of visual appearance andhiding pixels).

In some cases, modifying the presentation of the MR environment includesoccluding visibility to at least one user of an object in the MRenvironment when it is determined the occluder is positioned between theobject and the at least one user. FIG. 15 is illustrative of thisconcept in that the occluder for the person 1510 is occluding pixels ofthe hologram 1520, and the hologram 1520 is occluding the person 1515.Accordingly, method 1700 generally relates to a technique forconfiguring an occluder based on the assumed position of a person'sbody, where the assumed positioned is determined using pose dataacquired from a device used by the person.

Attention will now be directed to FIG. 18 , which illustrates aflowchart of an example method 1800 for dynamically updating anunderstanding of a user's body in a MR environment and for modifying anoccluder for the user, where the occluder is configured based on anassumed position of the user's body within the MR environment.

Method 1800 includes an act (act 1805) of determining an initialunderstanding of a user's body within a MR environment. In thisscenario, the assumed understanding of the user's body includes at leastan assumed position of the user's body, as has been described in detailthroughout the various figures.

In some cases, though not necessarily all, the initial understanding ofthe user's body includes a determination of the user's body positionwithin the MR environment based on an initial set of images/scans of theuser's body, as captured by one or more cameras of the user's HMD and/ora third-party device/system that scans the MR environment. For instance,perhaps an initial calibration, as discussed earlier, is conducted inorder to generate a baseline understanding of the user and/or theenvironment. In some embodiments, the process of updating theunderstanding of the user's body, where the update occurs subsequent tothe initial camera imaging or calibration, is then performed withoutperforming and/or relying on new scanning/imaging of the user's bodyand/or environment, and for at least a predetermined period of time.

Method 1800 then includes an act (act 1810) of presenting (optionallydisplaying but not necessarily displaying) an occluder within the MRenvironment, where the occluder corresponds with the initialunderstanding of the user's body and is based on the assumed position ofthe user's body within the MR environment. In some embodiments, as wasdescribed in FIG. 9 , the assumed understanding of the user's body maybe based on the relative position of the tracked device (e.g., an HMD)in the MR environment in combination with a determined relative positionof a second tracked device (e.g., a handheld physical device) in the MRenvironment.

Act 1815 involves identifying a relative position of a tracked devicethat is worn or carried by the user. Here, the tracked device comprisesa physical device. As described earlier, the tracked device may be anHMD, a handheld device/instrument, or perhaps a device worn or otherwisecarried by the user.

A change in a position or state of use of the tracked device is detectedin act 1820. In response, act 1825 involves updating the understandingof the user's body and corresponding assumption of the user's bodyposition within the MR environment based on the detected change inposition or state of use of the tracked device. Beneficially, thisoperation (and others) is performed irrespective of any camera imagingof the user's body. In some implementations, the process of updating theunderstanding of the user's body and corresponding assumption of theuser's body position within the MR environment is based on the detectedchange in state of use of the tracked device. For instance, in somecases, the change in state of use may comprise a change from active useto inactive use or, alternatively, a change from inactive use to activeuse.

Method 1800 then includes an act (act 1830) of modifying the occluderwithin the MR environment based on the update to the understanding ofthe user's body position. Modifying the occluder may be performed in anumber of ways, including changing a size, shape, or orientation of theoccluder. Additionally, modifying the occluder may be performed bychanging a configuration of the occluder from a first configuration to asecond configuration selected from any number of differentconfigurations of the occluder. In this example scenario, the secondconfiguration may be selected in response to a determination that thesecond configuration has a higher percentage of correspondence to anassumed position of the user's body in the MR environment based on thechange in state of use of the tracked device as compared to otherconfigurations of the occluder. FIG. 10 was representative of theseoperations in which an occluder, or at least a portion of an occluder,was selected because it had the highest level of correlation orcorrespondence to the user's assumed body position as compared to otheravailable occluders. Additionally, FIGS. 11 and 12 , which describedoperations corresponding to updating occluders based on movement and/orstate, generally relate to the operations described in method 1800.

Attention will now be directed to FIG. 19 , which illustrates aflowchart of an example method 1900 for dynamically updating a playercollider for a user within a MR environment based on an understanding ofthe user's body within the MR environment. Here, the player collidercomprises a defined region associated with the user's body, such as theplayer collider 1605 from FIG. 16 . As described earlier, a playercollider is a type of occluder that may or may not be modified invarious ways. For instance, the shape or size of the player collider maycorrespond with the shape and size of the user while in other cases theshape or size of the player collider may be an abstracted version orless granular version of a detailed occluder matching the size and shapeof the user.

Method 1900 includes an act (act 1905) of determining an initialunderstanding of a user's body within a MR environment. Notably, theassumed understanding of the user's body includes at least an assumedposition of the user's body.

In some optional cases, the process of determining the initialunderstanding of the user's body within the MR environment is based oncamera images of the user's body, such as perhaps during an initialcalibration process or perhaps during a subsequent or periodicre-synchronization process. As another option, the process ofdetermining the initial understanding of the user's body within the MRenvironment may be based on assumptions of use associated with thetracked device and sensor data that defines the relative position of thetracked device within the MR environment and without relying on cameraimages of the user's body within the MR environment.

Act 1910 involves presenting the player collider within the MRenvironment (but perhaps without rendering or displaying the playercollider in the MR environment), where the player collider correspondswith the initial understanding of the user's body and is based on theassumed position of the user's body within the MR environment. In someimplementations, the process of presenting the player collider doesinvolve rendering or displaying the player collider within the MRenvironment as a visual representation of the user's body. In somecases, an application executing in the MR environment (or perhaps isexecuting to provide the MR environment) will trigger a MR event when anobject (e.g., a hologram) in the MR environment and the player colliderspatially intersect or collide in the MR environment.

In some embodiments, the player collider is the occluder describedherein. In some embodiments, the player collider is a combination of theoccluder and an additional buffer region surrounding the occluder. Insome cases, the player collider is a box or rectangular prism that isformed based on the dimensions of the occluder and that behaves in amanner similar to the occluder. In some implementations, the occluderhas a detailed correlation to the size, shape, and outlines of theperson, but the player collider is less granular or less detailed and isgenerated to be more abstract relative to the occluder (e.g., perhapsthe player collider is formed of simple shapes, like cylinders, spheres,and so forth).

Optionally, the MR event includes determining that the object hasintersected with or collided with the user's body even when no collisionor intersection has actually occurred (e.g., the interaction is avirtual, not real, interaction between a hologram and a real-worldobject or other hologram). The application executing with the MRenvironment may then be updated to reflect that the user's body has beenintersected with or collided with the object. For instance, theholographic flame 1610 of FIG. 16 colliding or striking the playercollider 1605 is representative of the MR event described earlier.

Act 1915 involves identifying a relative position of a tracked devicethat is worn or carried by the user, where the tracked device comprisesa physical device. Then, there is an act (act 1920) of detecting achange in a position or state of use of the tracked device.

In response, method 1900 includes an act (act 1925) of updating theunderstanding of the user's body and corresponding assumption of theuser's body position within the MR environment based on the detectedchange in position or state of use of the tracked device. This update isbeneficially performed irrespective of any camera imaging of the user'sbody. For instance, the embodiments can prevent or refrain from scanningthe environment (for at least a period of time) in order to reducecompute and battery usage. Subsequently, act 1930 involves modifying asize or position of the player collider within the MR environment basedon the updating to the understanding of the user's body position.

Optionally, the method may further include an act of preventing ahologram rendered in the MR environment from intersecting the playercollider. For instance, if the hologram were to intersect the playercollider, or perhaps the user, then it may be the case that thephenomenon illustrated in FIG. 1 may occur, where the holographicglasses arms 110 and 115 appear when they should not. To prevent such ascenario, some embodiments may cause pixels that intersect the playercollider to be hidden, as illustrated by the lack of holographic glassesarms in the hologram 125 of FIG. 1 . In some cases, pixels intersectingor colliding with the player collider may trigger the appearance of newhologram content, such as perhaps a red flash to indicate the user hasbeen “hit” by the dragon's flame.

Accordingly, the disclosed embodiments are able to beneficiallyconfigure an occluder or player collider based on an assumptionregarding the size, shape, and orientation of a person holding orwearing a tracked device (e.g., an HMD, handheld instrument, etc.). Byperforming the disclosed operations, the embodiments are able to reduceexpenses associated with compute usage and are further able to reducebattery usage by refraining from continuously scanning the environment.

Example Computer/Computer Systems

Attention will now be directed to FIG. 20 which illustrates an examplecomputer system 2000 that may include and/or be used to perform any ofthe operations described herein. Computer system 2000 may take variousdifferent forms. For example, computer system 2000 may be embodied as atablet (2000A), a desktop or laptop (2000B), a wearable device (e.g.,HMD 2000C), a mobile device, a standalone device, or any otherembodiment as represented by the ellipsis 2000D. Computer system 2000may also be a distributed system that includes one or more connectedcomputing components/devices that are in communication with computersystem 2000.

In its most basic configuration, computer system 2000 includes variousdifferent components. FIG. 20 shows that computer system 2000 includesone or more processor(s) 2005 (aka a “hardware processing unit”), anoccluder generation engine 2010, and storage 2015.

Regarding the processor(s) 2005, it will be appreciated that thefunctionality described herein can be performed, at least in part, byone or more hardware logic components (e.g., the processor(s) 2005). Forexample, and without limitation, illustrative types of hardware logiccomponents/processors that can be used include Field-Programmable GateArrays (“FPGA”), Program-Specific or Application-Specific IntegratedCircuits (“ASIC”), Program-Specific Standard Products (“ASSP”),System-On-A-Chip Systems (“SOC”), Complex Programmable Logic Devices(“CPLD”), Central Processing Units (“CPU”), Graphical Processing Units(“GPU”), or any other type of programmable hardware.

The occluder generation engine 2010 may be implemented as a specificprocessing unit (e.g., a dedicated processing unit as described earlier)configured to perform one or more specialized operations for thecomputer system 2000, such as the methods recited herein. As usedherein, the terms “executable module,” “executable component,”“component,” “module,” or “engine” can refer to hardware processingunits or to software objects, routines, or methods that may be executedon computer system 2000. The different components, modules, engines, andservices described herein may be implemented as objects or processorsthat execute on computer system 2000 (e.g. as separate threads). Theoccluder generation engine 2010 (or perhaps even just the processor(s)2005) can be configured to perform any of the disclosed method acts orother functionalities.

Storage 2015 may be physical system memory, which may be volatile,non-volatile, or some combination of the two. The term “memory” may alsobe used herein to refer to non-volatile mass storage such as physicalstorage media. If computer system 2000 is distributed, the processing,memory, and/or storage capability may be distributed as well.

Storage 2015 is shown as including executable instructions (i.e. code2020). The executable instructions represent instructions that areexecutable by the processor(s) 2005 (or perhaps even the occludergeneration engine 2010) of computer system 2000 to perform the disclosedoperations, such as those described in the various methods.

The disclosed embodiments may comprise or utilize a special-purpose orgeneral-purpose computer including computer hardware, such as, forexample, one or more processors (such as processor(s) 2005) and systemmemory (such as storage 2015), as discussed in greater detail below.Embodiments also include physical and other computer-readable media forcarrying or storing computer-executable instructions and/or datastructures. Such computer-readable media can be any available media thatcan be accessed by a general-purpose or special-purpose computer system.

Computer-readable media that store computer-executable instructions inthe form of data are one or more “physical/hardware computer storagemedia” or “physical/hardware storage device(s)” that are distinguishedfrom and that exclude mere transmission or transitory media. Incontrast, computer-readable media that merely carry computer-executableinstructions are “transmission media”. Thus, the current embodiments cancomprise at least two distinctly different kinds of computer-readablemedia: computer storage media and transmission media.

Computer storage media (aka “hardware storage device”) arecomputer-readable hardware storage devices, such as RAM, ROM, EEPROM,CD-ROM, solid state drives (“SSD”) that are based on RAM, Flash memory,phase-change memory (“PCM”), or other types of memory, or other opticaldisk storage, magnetic disk storage or other magnetic storage devices,or any other medium that can be used to store desired program code meansin the form of computer-executable instructions, data, or datastructures and that can be accessed by a general-purpose orspecial-purpose computer.

Computer system 2000 may also be connected (via a wired or wirelessconnection) to external sensors (e.g., one or more remote cameras) ordevices via a network 2025. For example, computer system 2000 cancommunicate with any number devices or cloud services to obtain orprocess data. In some cases, network 2025 may itself be a cloud network.Furthermore, computer system 2000 may also be connected through one ormore wired or wireless networks 2025 to remote/separate computersystems(s) that are configured to perform any of the processingdescribed with regard to computer system 2000.

A “network,” like network 2025, is defined as one or more data linksand/or data switches that enable the transport of electronic databetween computer systems, modules, and/or other electronic devices. Wheninformation is transferred, or provided, over a network (eitherhardwired, wireless, or a combination of hardwired and wireless) to acomputer, the computer properly views the connection as a transmissionmedium. Computer system 2000 will include one or more communicationchannels that are used to communicate with the network 2025.Transmissions media include a network that can be used to carry data ordesired program code means in the form of computer-executableinstructions or in the form of data structures. Further, thesecomputer-executable instructions can be accessed by a general-purpose orspecial-purpose computer. Combinations of the above should also beincluded within the scope of computer-readable media.

Upon reaching various computer system components, program code means inthe form of computer-executable instructions or data structures can betransferred automatically from transmission media to computer storagemedia (or vice versa). For example, computer-executable instructions ordata structures received over a network or data link can be buffered inRAM within a network interface module (e.g., a network interface card or“NIC”) and then eventually transferred to computer system RAM and/or toless volatile computer storage media at a computer system. Thus, itshould be understood that computer storage media can be included incomputer system components that also (or even primarily) utilizetransmission media.

Computer-executable (or computer-interpretable) instructions comprise,for example, instructions that cause a general-purpose computer,special-purpose computer, or special-purpose processing device toperform a certain function or group of functions. Thecomputer-executable instructions may be, for example, binaries,intermediate format instructions such as assembly language, or evensource code. Although the subject matter has been described in languagespecific to structural features and/or methodological acts, it is to beunderstood that the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the embodiments may bepracticed in network computing environments with many types of computersystem configurations, including personal computers, desktop computers,laptop computers, message processors, hand-held devices, multi-processorsystems, microprocessor-based or programmable consumer electronics,network PCs, minicomputers, mainframe computers, mobile telephones,PDAs, pagers, routers, switches, and the like. The embodiments may alsobe practiced in distributed system environments where local and remotecomputer systems that are linked (either by hardwired data links,wireless data links, or by a combination of hardwired and wireless datalinks) through a network each perform tasks (e.g. cloud computing, cloudservices and the like). In a distributed system environment, programmodules may be located in both local and remote memory storage devices.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or characteristics. The described embodimentsare to be considered in all respects only as illustrative and notrestrictive. The scope of the invention is, therefore, indicated by theappended claims rather than by the foregoing description. All changeswhich come within the meaning and range of equivalency of the claims areto be embraced within their scope.

What is claimed is:
 1. A method for dynamically updating anunderstanding of a user's body in a mixed-reality environment andmodifying an occluder for the user associated with an assumed positionedof the user's body within the mixed-reality environment, the methodcomprising: determining an initial understanding of a user's body withina mixed-reality environment, the initial understanding of the user'sbody including at least an assumed position of the user's body;presenting an occluder within the mixed-reality environmentcorresponding with the initial understanding of the user's body andbased on the assumed position of the user's body within themixed-reality environment; identifying a relative position of a trackeddevice that is worn or carried by the user, the tracked devicecomprising a physical device; detecting a change in a position or stateof use of the tracked device; updating the initial understanding of theuser's body and corresponding assumption of the user's body positionwithin the mixed-reality environment based on the detected change inposition or state of use of the tracked device and irrespective of anycamera imaging of the user's body; and modifying the occluder within themixed-reality environment based on the updating to the initialunderstanding of the user's body position.
 2. The method of claim 1,wherein the initial understanding of the user's body includes adetermination of the user's body position within the mixed-realityenvironment based on camera captured imaging of the user's body in themixed-reality environment, and wherein updating the initialunderstanding of the user's body is performed without any camera imagingof the user's body.
 3. The method of claim 1, wherein the initialunderstanding of the user's body is based on the relative position ofthe tracked device in the mixed-reality environment in combination witha determined relative position of a second tracked device in themixed-reality environment, and wherein the tracked device is an HMD(Head Mounted Display) device that is worn by the user and the secondtracked device is a handheld physical device.
 4. The method of claim 1,wherein updating the initial understanding of the user's body andcorresponding assumption of the user's body position within themixed-reality environment based on the detected change in state of useof the tracked device, the change in state of use comprising a changefrom active use to inactive use or, alternatively, a change frominactive use to active use.
 5. The method of claim 1, wherein themodifying the occluder comprises changing a size or shape of theoccluder.
 6. The method of claim 1, wherein the modifying the occludercomprises changing a configuration of the occluder from a firstconfiguration to a second configuration selected from a plurality ofdifferent configurations of the occluder, the second configuration beingselected in response to a determination that the second configurationhas a higher percentage of correspondence to the assumed position of theuser's body in the mixed-reality environment, based on the change instate of use of the tracked device, than other configurations of theoccluder.
 7. A mixed-reality device configured to render a mixed-realityenvironment with an occluder within the mixed-reality environment andfor modifying the occluder based on an assumed positioned of the user'sbody within the mixed-reality environment, the mixed-reality devicecomprising: a display configured to render the mixed-realityenvironment, including the occluder within the mixed realityenvironment; one or more processors; and one or more storage deviceshaving stored computer-executable instructions that are executable bythe one or more processors for configuring the mixed-reality device toperform the following: determine an initial understanding of a user'sbody within a mixed-reality environment, the initial understanding ofthe user's body including at least an assumed position of the user'sbody; present the occluder within the mixed-reality environmentcorresponding with the initial understanding of the user's body andbased on the assumed position of the user's body within themixed-reality environment; identify a relative position of a trackeddevice that is worn or carried by the user, the tracked devicecomprising a physical device; detect a change in a position or state ofuse of the tracked device; update the initial understanding of theuser's body and corresponding assumption of the user's body positionwithin the mixed-reality environment based on the detected change inposition or state of use of the tracked device and irrespective of anycamera imaging of the user's body; and modify the occluder within themixed-reality environment based on the updating to the initialunderstanding of the user's body position.
 8. The mixed-reality deviceof claim 7, wherein the initial understanding of the user's bodyincludes a determination of the user's body position within themixed-reality environment based on camera captured imaging of the user'sbody in the mixed-reality environment by a camera of the mixed-realitydevice, and wherein updating the initial understanding of the user'sbody is performed without any camera imaging of the user's body.
 9. Themixed-reality device of claim 7, wherein the initial understanding ofthe user's body is based on the relative position of the tracked device,comprising a first tracked device, in the mixed-reality environment incombination with a determined relative position of a second trackeddevice in the mixed-reality environment, and wherein the first trackeddevice is incorporated into the mixed-reality device, the mixed-realitydevice comprising an HMD (Head Mounted Display) device that is worn bythe user and the second tracked device is a handheld physical device.10. The mixed-reality device of claim 7, wherein updating the initialunderstanding of the user's body and corresponding assumption of theuser's body position within the mixed-reality environment based on thedetected change in state of use of the tracked device, the change instate of use comprising a change from active use to inactive use or,alternatively, a change from inactive use to active use.
 11. Themixed-reality device of claim 7, wherein the modifying the occludercomprises changing a size or shape of the occluder.
 12. Themixed-reality device of claim 7, wherein the modifying the occludercomprises changing a configuration of the occluder from a firstconfiguration to a second configuration selected from a plurality ofdifferent configurations of the occluder, the second configuration beingselected in response to a determination that the second configurationhas a higher percentage of correspondence to the assumed position of theuser's body in the mixed-reality environment, based on the change instate of use of the tracked device, than other configurations of theoccluder.
 13. A method for dynamically updating a player collider for auser within a mixed-reality environment based on an understanding of theuser's body within the mixed-reality environment, the player collidercomprising a defined region associated with the user's body, the methodcomprising: determining an initial understanding of a user's body withina mixed-reality environment, the initial understanding of the user'sbody including at least an assumed position of the user's body;presenting the player collider within the mixed-reality environmentcorresponding with the initial understanding of the user's body andbased on the assumed position of the user's body within themixed-reality environment, wherein an application executing in themixed-reality environment will trigger a mixed-reality event when anobject in the mixed-reality environment and the player colliderspatially intersect or collide in the mixed-reality environment;identifying a relative position of a tracked device that is worn orcarried by the user, the tracked device comprising a physical device;detecting a change in a position or state of use of the tracked device;updating the initial understanding of the user's body and correspondingassumption of the user's body position within the mixed-realityenvironment based on the detected change in position or state of use ofthe tracked device and irrespective of any camera imaging of the user'sbody; and modifying a size or position of the player collider within themixed-reality environment based on the updating to the initialunderstanding of the user's body position.
 14. The method of claim 13,wherein the mixed-reality event includes determining that the object hasintersected with or collided with the user's body even when no collisionor intersection has actually occurred and updating an application thatis executing with the mixed-reality environment to reflect that theuser's body has been intersected with or collided with the object. 15.The method of claim 13, wherein the method further includes preventing ahologram rendered in the mixed-reality environment from intersecting theplayer collider.
 16. The method of claim 13, wherein the method furtherincludes presenting the player collider without rendering the playercollider in the mixed-reality environment.
 17. A mixed-reality deviceconfigured for dynamically updating a player collider for a user withina mixed-reality environment based on an understanding of the user's bodywithin the mixed-reality environment, the player collider comprising adefined region associated with the user's body, the mixed-reality devicecomprising: a display; one or more processors; and one or more storagedevices having stored computer-executable instructions that areexecutable by the one or more processors for configuring themixed-reality device to perform the following: determine an initialunderstanding of a user's body within a mixed-reality environment, theinitial understanding of the user's body including at least an assumedposition of the user's body; present, on the display of themixed-reality device, the player collider within the mixed-realityenvironment corresponding with the initial understanding of the user'sbody and based on the assumed position of the user's body within themixed-reality environment, wherein an application executing in themixed-reality environment will trigger a mixed-reality event when anobject in the mixed-reality environment and the player colliderspatially intersect or collide in the mixed-reality environment;identify a relative position of a tracked device that is worn or carriedby the user, the tracked device comprising a physical device; detect achange in a position or state of use of the tracked device; update theinitial understanding of the user's body and corresponding assumption ofthe user's body position within the mixed-reality environment based onthe detected change in position or state of use of the tracked deviceand irrespective of any camera imaging of the user's body; and modify asize or position of the player collider within the mixed-realityenvironment based on the updating to the initial understanding of theuser's body position.
 18. The mixed-reality device of claim 17, whereinthe mixed-reality event includes determining that the object hasintersected with or collided with the user's body even when no collisionor intersection has actually occurred and updating an application thatis executing with the mixed-reality environment to reflect that theuser's body has been intersected with or collided with the object. 19.The mixed-reality device of claim 17, wherein the computer-executableinstructions are further executable by the one or more processors forconfiguring the mixed-reality device to prevent a hologram rendered inthe mixed-reality environment from intersecting the player collider. 20.The mixed-reality device of claim 17, wherein the computer-executableinstructions are further executable by the one or more processors forconfiguring the mixed-reality device to present the player colliderwithout rendering the player collider in the mixed-reality environment.